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ABSTRACT 

Findings from a study of Illinois' educational 
evaluation process, in which external evaluators conduct the 
evaluation, are presented In this paper. Methodology Involved 
analysis by State Board of Education staff of requests for proposals 
(RFPs) ; these were mailed to 121 potential bidders for external 
evaluation. Bidders Included universities, individual consultants, 
private nonprofit and for-profit companies, and regional research 
laboratories. Application of systematic standards to the proposals 
resulted in the consistently higher ran)cing of the for-profit 
corporations over universities or non)>roflt corporations. Findings 
indicate that a thorough exeunlnatlon of researchers' credentials is 
needed, as well as quality controls and monitoring that focus on 
staff l.ig, technical design, time frame, anri the specification of 
expecced outcomes. A conclusion is that for-profit corporations are 
more IDcely to respond to the needs of the funding sourc«> within the 
time frames Involved. Three tables are Included. Appendices include 
the request for proposal process, reform evaluation results, and 
results of applying standards. (LMI) 
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Assessing Educational Reform: Illinois Experience 



Education Reform 

During the 1980s, the nation observed the most thorough and 
sustained effort to reform the American public educational system 
(ISBE, 1985). The release of A Nation At Risk in 1983, brought 
to a pinnacle the growing concern among those who have turned to 

the schools to benefit their children and grandchildren 

and found them needing improvement. The education reform 
movement spread natiorwide and in Illinois, major legislation 
intended to help schools improve was passed in 1985. According 
to the Education Commission of the States, it put Illinois at the 
"head of the class" in school reform nationwide. The result of 

that legislative session the Education Reform of 1985, 

contained 169 individual components comprising a comprehensive, 
far reaching approach to strengthening and improving educational 
opportunities for children. 

The 169 separate e.'cements of the legit:lcitive package are most 
easily examined when categorized under six themes: A Nev.' View of 
Schooling, Children-At-Risk, the Quality of Educational 
Personnel, A New Role for Principals, Reorganization, and the 
Work Continues. A common and consistent thread is woven 

throughout the fabric of these thematic areas a clearly 

defined emphasis on student learning. While the new laws 
approach school improvement from a variety of directions, they 
all eventually focus on the same aestion: when this change is 
made, what benefits will ^'e realized for improving students' 
opportunities to learn ani, through them, improving student 
achievement . 

The vision of the General Assembly and the Governor in approving 
the Education Reform of 1985, was to revitalize elementary and 
secondary schools in Illinois. The ensuing years brought about 
the development and implementation of numerous program 
init iat ives . 

Reform Evaluation 

Five years after the passage of the 1985 reform, and in response 

to an Illinois State Board Goal To Evaluate Educational 

Reform Efforts in Illinois and Determine Areas in Which Changes 

and/or Additional Efforts Are Needed a systematic evaluation 

effort was developed and implemented. Money appropriated by the 
Illinoxs General Assembly for t^e evaluations carried with it the 
stipulation that an external evaluator conduct the efforts. 
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A point worth noting here is that a contractor could, and in fact 
was awarded more than one contract over the three fiscal years. 
In fact, the entities awarded contracts consisted of one (1) 
university ~ different departments, one (1) not-for-profit 
corporation - with different contract managers and one (1) for- 
profit corporation - included same project managers and staff. 
For specific descriptive information on the program and 
highlights of the evaluation, se^ Appendix B. 

Fiscal constraints brought to light the need to examine the 
utility and quality of the evaluation efforts conducted by the 
external contractors. Those who had been awarded contracts were 
considered to be competent, reliable sources by the proposal 
reviewers, it was therefore necessary, to make determinations 
regarding credibility, using other measures. 



Using sections from The Standards for Evaluation of Educational 
Programs, Projects and Materials (1988), (Appendix C) , an effort 
was initiated to enable the Illinois State Board of Education to 
make judgments about the credibility of the evaluation reports. 
These judgments are based primarily on the information provided 
in the evaluation reports and, to some extent, on the contractors 
original proposals. Using a team approach, three reviewers 
scored the evaluation reports on their adequacy to meet the 
Standards. Before the scoring occurred, all labels and symbc.s 
were removed in order to remove the identify of the contracto/.- . 
A yes scc-e received a one (even yes's with qualifications) ^ •And 
a no received a score of zero. Since selected portions of T;he 
standards were used, a proposal could receive a score betwetm 0 
and 15. Consensus agreememt was required before a standard score 
could be gj -en for each standard. 

The fol.lowinq table provides the results of applying the 
Standards . 
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Table 2 



Results of Applying Standards for Evaluation 
of Education Programs, Projects, and Materials by Evaluation 



The two reports receiving the lowest scores were the efforts of 
the same university, but the individuals conducting the 
evaluations were from different departments within the 
university. 

A better display of the results of applying the Standards can be 
found in Table 3 . 



Program Area 



Total Score 



Educational Service Centers 
Reading Improvement 
Administrators' Academy 
Prekindergarten At-Risk 
Report Card 
Staff Development 
Bilingual 

Gifted/Remedial Summer School 



6 

14 

15 
16 
7 

15 
10 
14 



Tabla 3 



Results of Applying the Standards for Evaluation of 
Education Programs, Projects and Materials by Contractor 



Contractor 



Results 



University 
A 
B 



6 

7 



For 



Profit Corp. 



A 
B 
C 



14 
15 
15 



Not 



For-Profit corp. 



A 
B 

C 



15 
10 
14 



Applying the Standards to the evaluation efforts conducted by the 
For-Prof it-Corporation, resulted in consistently high scores for 
all three efforts. Standards where zero scores were given 
related to sufficient scope of information collection and sources 
of information. 

The results of applying the standards to the evaluations 
conducted by the Not-for-Prof it-Corporation were consistently 
high in two efforts. The two evaluation efforts receiving the 
high scores were conducted by the same projtect manager and staff. 
The low scoring effort was conducted by a different project 
manager and the evaluation effort was staffed by consultants from 
other states. Standards where zero scores were given related to 
information collection, interpretations of findings, program 
descriptions, disclosure of information, conclusions, and 
evaluation of safeguards. 

The evaluations conducted by university contractors scored 
consistently low (6 and 7 out of 15) , when the standards were 
applied. The specific stand^irds where a zero score was awarded 
are noted as follows: 

A3: Was the information collected, of sufficient scope and 
selected in such ways to address the RFP questions and meet the 
needs and interest of ISBE? 



I 



COMMENTS: In both of the evaluation et'fortif conducted by the 
university - reviewers noted that the RFP questions were not 
fully answered. Nor did the contractor explain or provide 
information as to why the questions were not adequately and 
completely addressed. Other areas of concern related to the 
limited and somewhat questionable lack ot inclusion of certain 
stakeholders. For example, the evaluation design for effort B 
included the use of broad based focus groups composed of 
stakeholders. These groups were to provide information for the 
development of a survey and other information collection 
activities. When implemented, one of the most critical 
stakeholder groups - parents - was limited to the city closest to 
the university. The parents included in the focus group were 
university associated and did not represent the average or 
special parent. By not seeking information from a broader 
audience, the results from the focus group were biased. 



A4: Were the perspectives, procedures, and rationale used to 
interpret the findings carefully described so that the basis for 
value judgments are clear? 

COMMENTS: The contractor seemed to put more weight on interview 
and case study findings when survey results contradicted the 
findings. The contractor made no attempt to clarify or justify 
this issue. 

AS: Did the evaluation report describe the program and its 
context, the purpose, procedures and findings of the evaluation, 
so that the funding agency (ISBE) could readily understand what 
was done and why it was done, what information was obtained, what 
conclusions were drawn, and what recommendations were made? 

COMMENTS: The contractor did not provide an accurate description 
of the program area associated with the evaluation B effort. 
References to specific data elements comprising the program were 
inaccurate and therefore, the findings and results were 
inaccurate. For example, the contractor stated that the Illinois 
School Report Card contains dropout rates. The contractor had 
copies of the report card, the legislation and supporting 
information, which clearly stated what the report card contains. 
The dropout rate is not a part of the school report card. 

Additionally, at least one of the recommendations did not have 
supporting documentation to back its existence. Other 
recommendations called for reducing the information in the report 
card while another recommendation called for expanding the 
information . 



A7t Was the report released in a tinely manner so that ZSBE 
could best use the reported information? 

The report or final products for their evaluation effort was due 
on April 30th/ but was not received until June. ISBE had 
originally intended to use the evaluation reports in making 
programmatic revisions and amending legislation. Due to late 
receipt of the reports, we were unable to do either. 



C3 Was the written evaluation open, direct, and honest in its 
disclosure of pertinent findings, including the limitations of 
the evaluation? 

Yes, with qualifications. The contractors did briefly describe 
the limitations of the survey findings for administrators, 
teachers and parents, as well as nonresponse rates. They 
described the uses of the focus groups, interviews, and case 
studies. They failed, however, to describe any limitations to 
these last three procedures, such as respondent bias or lack of 
knowledge about the program. For example, one case study 
included an administrator's comments about the accuracy of the 
dropout data on the Report Card. There are no dropout figures on 
the Report Card. The authors failure to recognize these 
instances hurts the credibility of the case studies in the 
report. 

C7: Was the evaluation complete and fair in its presentation of 
strengths and weaknesses of the Report card so that the strengths 
can be built upon the problem areas addressed? 

No, with the qualification given in C3 . 



D2 Was the context in which the Report Card is used examined in 
enough detail so that influences on it can be identified? 

Yes, with qualifications. For example, the contractors case 
study descriptions to provide a context in which the Report Card 
is used. Further, the contractors response to RFP questions 
particularly, illustrates how the popular media's use of the 
report card can have indirect effects of school activities. 

DIO Were the conclusions reached in the evaluation explicitly 
justified, so that ISBE can assess them? 

No, for the reason specified in C3 . 

Dll Did the evaluation procedures provide safeguards to protect 
the evaluation findings and report against distortion by one's 
personal feelings and biases of any party to the evaluation? 



No, the level of knowledge about the Report Card and the 
Educational Service Centers by respondents in the interviews and 
focus groups was not addressed in the report. This affected the 
credibility of the case study, interview, and focus group 
findings. 



Implications 



There are several implications that can be drawn from this 
exercise. These are noted as follows: 

Agency Perspective : As with any effort requiring a written 
product, contracts are often awarded on the basis of well written 
proposals that specifically address the required questions being 
asked. The Illinois State Board of Education's experience over 
the last three years, leads us to believe that there is a vast 
difference between well written proposals and the actual 
implementation of the design described in the proposal and the 
final product or outcome. This awareness allows for certain 
quality control efforts to be put in place. 

Proposal Review : The issue of who will be conducting the study 
appears to have a relationship to the quality of the expected 
product. The Request For Proposal requires respondents to 
include the names and qualifications of the proposed project 
manager and staff. Revised review criteria will provide for a 
more thorough examination of the credentials of the individuals 
who will staff this study. 

Monitoring of Contract ; Closer and consistent monitoring of the 
contract in v.he way of frequent communication via meetings and 
progress reports will provide a vehicle for alerting us to 
problem areas, such as delays, staff limitations, etc. 

The monitoring activities will focus on specific areas such as: 

Staffing ; Changes from the original proposal in 
who will staff the study will have to be approved 
by the Illinois State Board of Education. 

Technical Design : All data and information collection 
methods and audiences will have to be approved by the 
Illinois State Board of Education prior to use. This 
should help eliminate issues regarding scope of the 
information being collected and potential bias issues. 
Additional efforts will focus on who (stakeholders) will 
participate in the study. The intent will be to try to 
avoid narrowly defined focus groups and samples. 

Time Frame ; The contracts awarded over the last 
three years limited the time frame for the evaluation 
studies to 7-8 months. The time frame is dictated by 
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budgeting requirements and the need to have information 
for any legislative changes by early spring. Several 
of the final evaluation products were not received within 
the contractual time frame. The late submission resulted 
in the inability to use the findings and recommendations 
to make legislative changes* In these two instances, 
the late submission of a final product defeated one of the 
purposes of conducting the evaluation. Earlier release of 
RFP's and selection of contractors should help with issues 
related to lack of time. 

Specifvina Expected Outcomes ; It is evident that the 
Illinois state Board of Education will have to be more 
active in identifying the expected outcomes of programs. 
One thing that handicapped the contractors was the lack 
of specific stated outcome (s) for a program or lack of 
available data at the site level to support program 
outcomes . 



Author Comments : It is evident from the results of this 
exercise, that those whose livelir.ood depends on producing 
acceptable products are more likely to conduct the studies as 
stated in proposals, and produce timely products. Those whose 
financial support is not solely contractual, may approach 
conducting evaluation studies from a different frame of mind. 
Our experience with this type of contractor leads us to believe 
that they are less responsive to the needs of the funding source 
'.nd the time frames involved. 
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Appendix A 

- Goal of the Reauaat for Proposal 

This statement provides the potential bidder 
with a clearly stated, direction-oriented, 
goal(s) of the RFP. 

Time/Fiscal Parameters 

This section must outline two (2) criteria 

a) Time span of the proposed contract 

(e.g. term of contract is March 1, 

through June 30, ) . 

b) Monetary terms of contract (e.g., amount 

requested not to exceed $ ; amount for 

subcontractor. 

~ Evaluation Design 

The proposal must contain an appropriate design (s) for 
evaluation of effectiveness of the proposed project. 
Types of data to be analyzed and criteria must be 
included . 

- Budget 

The proposed budget must be presented in terms of specific 
objectives and cross-ref erar by line item of 
expenditure . 

- Subcontractors 

The following information is required if any 
subcontracting is to be utilized: 

1) Name and address of subcontractor (s) if 
known 

2) Nature of contract (s) 

3) Measurable and time-specific productf! 
expected 

4) Associated costs 

- Project Personnel 

The following information concerning project personnel 
must be provided; 
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1) Project Director 

a) Experience with activities similar 
to those in this RFP 

b) Experience in directing major 
projects, including duration of 
projects, similar to that requested 
in this RFP 

c) Names, addresses, and telephone 
numbers of three persons, references, 
knowledgeable regarding past performance 

2) Project Staff 

Description of similar programs and projects, 
including duration, for individual project 
staff members 

3) Staff Limitations 

Limitations the program staff would have 
related to: 

a) Travel 

b) Length of time available to be contracted 

c) Earliest starting date 

4 ) Other Resources 

Other resources, such as consultants, agencies, 
etc., from which the potential bidder can drav, 
not to exceed % of total amount) 

Proposal Specifications : 

a. Proposal Objectives ; Objectives of proposed project 
must be stated in measurable terms. 

b. Proposed Activities : Activities relating to each 
objective must be detailed in a time-specific format. 
Flow charts should be included. 

GENERAL INFORMATION 



Payment : 



Rights: 



Payment for services and expenses 
will be made on a reimbursement of 
cost basis . Dates of payment will 
be negotiated to insure that this 
office pays only for completed 
performance . 

The Illinois State Board of Education 
assumes rights to all mr":erials 
developed in this project. 
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Letter of Intent: (Note: A letter of intent is optional 

and is requested by the author of the 
RFP only if that person desires to do 
sc. If requested, the RFP must indicate 
that "the letter of intent to submit a 
proposal should be forwarded no later 
than ."). 

Statement of Assurances: The attached Statement of Assurances 

concerning compliance with state and 
federal regulations must be signed. 
(Note: The appropriate statement is 
available through the Planning/Policy 
Analysis Section) . 

Compliance with Data 

Collection Procedures: All data collection activities to be 

carried out within the contract must 
have prior approval of form, instrument- 
ation and content by the Illinois State 
Board of Education. Approval may be 
obtained from the Illinois State Board 
of Education, Office of Data Management 
Administration . 

Past Contracts 

Completed: Proposals will be accompanied by a list 

of all major contracts completed in the 
last three years in related areas of 
expertise stating: 

a) Monetary amount of each contract 

b) Date each contract was completed 

c) Elapsed time from each contract date 
to completion and submission date 




Criteria : 



1. Soundness of Approach ; Emphasis here is on the evalua- 
tion design, including instrument selection/development, 
sampling (if appropriate), techniques for collecting and 
analyzing data, sequence and relationships of major 
steps, and methods for managing the contract. 

2. Qualifications of Firm and Staff : Th: : criterion 
includes evidence of training, prior relevant experience 
and the ability of the contractor to meet the terms and 
time constraints of this RFP. 

3. Adherence to RFP ; The degree to which all specifica- 
tions of the RFP are followed. 



4. Cost-Effectiveness ; The reasonableness of proposed 
charges in view of the Technical Plan (design) , 
objectives, activities/execution, products, staff 
qualifications and staff time allocation. 



Process ; 

1. The Illinois State Board of Education reserves the right 
to reject any or all proposals submitted in response to 
this RFP. 



2. The Illinois State Board of Education is not required to 
select a contractor solely on the basis of the lowest 
bid. The Illinois State Board of Education will 
negotiate with the successful bidder to assure satis- 
factory costs and services. 

3. The Illinois State Board of Education reserves the right 
to negotiate the final terms of the contract. All 
aspects of the approved proposal from the successful 
applicant will become contractual obligations. 



Proposal Copies: Ten copies of the proposal must be 

received by the contact person specified 
in this RFP by the close of the ISBE 

business day (4:30 p.m.) on 

(Note; More than 10 copies can be 
required if necessary) . 

contact Person 



Department 
Address 



city/State/Zip Code 



Telephone 



Project Period: 



Proposed project must be for the period 

to . (Note: Potential bidders 

should be allowed at least three weeks 
to develop the proposal. At least two 
weeks s wuld be allowed for internal 
ISBE review) . 
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Appendix B 
Reform Evaluation Results 



FY 90 

Programs evaluated: 

* Reading Improvement Program 

* Educational Service Centers 

The Reading Improvement Program was evaluated by an out-of-state 
for-profit corporation. The findings of the evaluation reveal 
that 75% of all kindergarten through sixth grade students 
received services through the Reading Improvement Program. 
Tremendous diversity exists among schools in how programs are 
structured, with program design reflecting local school district 
needs. Many programs are targeted at specific grade levels, the 
preponderance of these are in kindergarten through second grade. 

Thirty-five to forty percent of the students served by the 
Reading Improvement Program are not receiving reading services 
from any other source. In general, students are perceived to be 
performing better in reading and the benefits which result from 
the Reading Improvement Program are worth the implementation. 

The Educational service Center Program was evaluated by an 
Illinois university. The results of the evaluation indicated 
that Educational Service Centers in Illinois provide a wide 
variety of services to local school districts and their personnel 
and, in general, ha"e been successful. Strengths were cited in 
areas of inservice education and staff development. Particular 
areas of noted strength include services targeted to the 
Administrators' Academy and school improvement initiatives 
related to educational reform. 

Weaknesses include the lack of clarity about their purpose, lack 
of consistency across the state in quality of service, and 
political turmoil surrounding their governance and operation. 

FY 91 

Programs evaluated: 

* Illinois School Report Card 

* Illinois Administrators' Academy 

* Illinois Prekindergarten At-Risk Program 
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The Illinois School Report Card Program was evaluated by an 
Illinois university. The results show there exists substantial 
support among parents and other stakeholders for the School 
Report Card as an accountability measure, a means to compare 
schools, and a process and document for promoting increased 
discussion about schools. The requirement that the Schoo:. Report 
Card data be made available to local media has helped inform 
parents about their school and other schools in their area. 
There is some evidence that changes ".re taking place in schools 
as a result of the School Report Card, but most of the changes 
cited were intangible and difficult to measure. 

Information identified as being most important by stakeholders 
include average class size, pupil/teacher ratio, district type 
and average size, per-pupil expenditures, breakdown of district 
finances by funds, student attendance rate, graduation rate, 
nationally-normed achievement information and student mobility. 

The most problematic student information item is the Illinois 
Goal Assessment Program (IGAP) test scores. The study found 
little evidence that the IGAP test information was considered 
important either by parents or by school practitioners. The most 
frequently cited recommendations were to make the report card 
easier to understand, to reduce jargon and technical terms, use 
graphs and charts where possible, and improve the format and use 
of space. 

The Illinois Administrators' Academy Program was evaluated by a 
not-for-profit corporation. The findings indicated that 
administrators throughout the state perceive the Academy to be an 
important link in facilitating the role of the principal as 
instructional leader and improving teacher performance. 

Academy strengths include regional involvement, the quality of 
Illinois State Board of Education trainers and program support, 
reasonable costs, convenient training locations, flexible 
scheduling, timely and appropriate program topics, and trainers 
who are experienced practitioners. 

Participation among principals and superintendents in the 
se]ective strand of the program continues to grow with 
substantial numbers electing to participate in the advanced 
training phases of the academy. 

Recommendations include preparing general operational guidelines 
for the Academy to reflect developmental changes that have 
occurred since 1986, developing standardized data collection 
procedures, encouraging inter-ESC cooperation, providing a 
broader range of academy services to superintendents, and 
continuing to provide training for local administrators to foster 
positive educational change within regions. 
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The Illinois Prekinderaarten At~Risk Program was evaluated by a 
for-profit corporation. The findings of the evaluation indicated 
that the program has been implemented as indicated in the law and 
the State Board of Education regulations. Programs are diverse 
and permit local needs to determine local services. The 
appropriate target population, children at-risk of academic 
failure, is being served. In the variety of programs that exist, 
developmentally appropriate practices have been implemented. The 
programs have similar philosophies and address the total child 
physically, emotionally, socially, and cognitively. Programs are 
implementing a variety of parent involvement activities wich 
parent participation being high for the populations being served. 

Reports of program success are similar across all projects and 
include assessment in valid outcome measures of readiness for 
kindergarten, promotion rates, special education and supplemental 
program placement, and student performance by subject area. Well 
over half (70% - 80%) of prekindergarten children followed-up for 
one, two, and three years are reported to have average or above 
average performance in reading, mathematics, language and 
behavior . 

Recommendations include the continued support and promotion of 
flexible local programs, cooperative joint agreements, and 
interagency - inter-program collaboration, and continued increase 
in the allocation of funds to permit the program to expand to the 
districts and to serve all identified at-risk children without 
diminishing the funds currently expended. 

FY 92 

Programs evaluated include: 

* Bilingual Education 

* Staff Development 

* Gifted/Remedial 

Highlights from the Evaluation of the Illinois staff Development 
Program 

The Illinois Staff Development Program was established by the 
1985 Illinois Reform Act (Section 2-3.59, 2-3.60, School Code) 
and requires staff development program development and 
implementation from all public school districts throughout the 
state. Each district that submits a plan receives funding based 
on their number of full-time equivalent certified instructional 
staff. During the past six years, the average annual 
appropriation to this program has been $3,000,000.00. 
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The existing Illinois Staff Development Program 
planning model designed by ISBE is a theoretically 
sound model and is of great practical value in 
guiding local districts in planning staff develop- 
ment programs. 

Teachers and administrators perceive staff develop- 
ment programs as an effective means of improving 
student outcomes and classroom instruction. 

The staff development plans examined reflect the 
shifting trend in staff development from the 
traditional top down approach to a more "grass 
roots" or school-based .effort . 

A continuum of staff development opportunities 
is available in Illinois for teacher preparation 
from one-day institutes to planned ongoing efforts 
previously sponsored by local districts and ESCs. 

Different models of delivery are used by the various 
providers of staff development activities. ESCs 
usually offer workshops that vary in length from two 
hours to multi-session opportunities. ESRs typically 
offer only single-day institutes; some sponsor 
regional conferences in addition to or in lieu of 
institute days. Some local districts may use all of 
the models of delivery while others may only participate 
in regionaJ. institutes. 

The majority of teachers indicate that their needs for 
staff development have changed over the past three 
years . 

Teachers noted lack of time as the primary factor in 
inhibiting their attendance at staff development 
programs. 

The focus for planning and implementing staff develop- 
ment programs has been appropriately placed at the 
local level; however, sufficient technical assistance 
is not available from ISBE nor received from ESCs 
for developing quality plans and evaluation procedures. 

The level of cooperation between agencies depends on 
the governance structure that exists between the ESR, 
the ESC, and the local district. Cooperation ranges 
from very high to competitive, to counter-productive. 
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Key RacomiT^endation 

* Adequate funding of staff development programs in the 
State of Illinois is needed in order to improve the 
quality and quantity of services that are currently 
being promoted. 

* Leadership in the area of professional development is 
needed from all levels of education. The ISBE, local 
districts and building administrators should make 
professional development for the teaching pool a 
priority. 

* Local districts should use methods for identifying 
needs that are based on the gaps between where schools 
and districts are and where they want to be in terms 
of student, teacher, school, and district variables. 

* To eliminate duplicative effort and administrative 
costs, all responsibilities for staff development and 
associated funding currently assigned to the ESRs in 
the Illinois School Code should be transferred to the 
ESCs. 

* Regional responsibilities for staff development should 
be consolidated into the ESCs, including 
responsibilities for Special Education and Vocational 
Education. 

* The current ISBE staff development program planning 
model and application process should be continued with 
some minor modifications. 



Highlights from the Evaluation of Bilingual Education Programs 

Bilingual education programs were established in Illinois in 
recognition of the fact that instruction only in English is often 
inadequate for children whose native tongue is another language. 
In order to ensure equal educational opportunity for all 
children, the Illinois General Assembly has mandated that either 
a Transitional Bilingual Education Program or a Transitional 
Program of Instruction be provided for K-12 limited-English 
proficiency students in order to assist them in achieving grade 
level proficiency in English. The FY 92 state appropriation for 
bilingual education programs was $49,354,900.00. 

Key Findings 

* A wide variety of instructional techniques is in 
use but the pull-out model is very prominent, 
especially in lower grades. 
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* There is a consensus among administrators and program 
staff that effective programs are characterized by a 
high quality staff, well defined services, integration 
with mainstream curriculum, emphasis on reinforcing 
student self esteem, and maintaining flexibility within 
a rapidly changing environment. 

* Administrators who identify their programs as effective 
say that they are sufficiently funded, have public 
awareness and participation, have adequately trained 
staff, and emphasize individual student needs. 

* Bilingual education personnel perceive that the 
educational experience of students who have exited the 
programs tends to be satisfactory, unless the student 
lacks basic academic skills. 

* The greatest factors cited by education administrators 
as limiting the quality of transitional programs are 
limited state funding, lack of qualified staff, and 
lack of continuing staff development opportunities. 

* Transitional programs are coordinated with mainstream 
education, but coordination with other services such 
as Chapter 1 and Special Education is less likely due 
to lack of time, lack of bilingually sensitive staff 
to deliver these services, and a concern for over- 
emphasis of these special service areas at the expense 
of the student's total educational experience. 

Key Recommendations 

* Increase funds to provide more staff development 
opportunities for both bilingual and mainstream 
teachers and administrators. 

* Issue guidelines for integrating bilingual education 
programs with Chapter 1, Special Education, and Gifted. 

* Issue guidelines regarding recommended transition 
criteria tied to student outcome?, rather than a state 
statutory limit in terms of years served. 

* Establishing procedures to formally track the progress, 
both academic and social, of currently served and of 
transitioned students. 

* Extend fu.iding to provide services (including tutoring, 
counseling, and supplementary English-language 
instruction) to students who have transitioned into 
mainstream programs. 

* To the extent practicable, emphasize keeping language 
minority students in their "home" schools. 
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* Consider revising the state's position on the use of 
Title VII funds, especially when funding levels are 
being reduced. 

Highlights from the Evaluation of Summer School Gifted and 
Remedial Programs 

The Summer School Gifted and Remedial Program was established as 
part of the educational reform legislation enacted by the 
Illinois General Assembly during its 1985 session. The program 
provides qualified districts across the state with grants to 
conduct a summer school program for students who are gifted, 
talented, or in need of remedial education, without having to pay 
tuition, fees, or instructional material expenses. The program 
first became operational in the summer of 1986. The FY 92 
appropriation for the Summer School Gifted and Remedial Program 
was $15,000,000.00. 

Key Findings 

* The major advantages cited by local administrators 
for offering the program were the flexibility of the 
program in accommodating districts needs, the cost 
efficiency of the course offerings vis-a-vis repeating 
a grade, and the varied instructional activities that 
stimulated student attendance. 

* The factors identified by superintendents, principals 
and teachers as having the greatest positive impact on 
the operation of the program were the teacher-student 
ratio, parents' attitudes toward the program, curriculum 
materials, and the opportunities for students to use 
computers. The uncertainty of state appropriations was 
identified as having a negative impact. 

* Local school administrators and teachers identified the 
following positive outcomes for students who partici- 
pated in the program: a) student grade retention had 
decreased, b) academic performance had improved, 

c) students were better prepared to do their school 
work, and d) student attitudes toward school had 
improved. 

* Local school administrators and teachers indicated that 
the program could be improved by: a) increasing coordi- 
nation between summer school and regular classroom 
teachers, b) obtaining parental support for the program, 
c) getting the students of greatest need into the 
program, d) promoting good student attendance, e) pro- 
viding qualified teachers, and f) providing student 
transportation . 
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* Eighty-three percent of the funds for the program were 
targeted to remedial programs and the remaining 17% 
were used for gifted programs. The allocations of thes 
funds were 83% for certificated personnel salaries, 10% 
for instructional materials, 5% for pupil transporta- 
tion, and the remainder for students tuitioned out to 
other districts or outside agencies. 

Key Recommendations 

* Encourage smaller districts to enter in a multiple 
district program in order to provide a broader range 
of programs to their students and to improve cost 
effectiveness . 

* Encourage Educational Service Centers to provide 
services to districts relevant to the summer school 
program, such as assessment and coping with problems 
identified by districts. 

* Revise the ISBE grant application form and schedule for 
submission to facilitate program monitoring and utility 
of the program database in generati.ng timely reports. 

* When funds become available for the summer school 
program, make grant information available to districts 
as soon as possible so that they may proceed with 
program planning. 
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Appendix c 
Results of Applying standards 



Al Were the audiences involved in or affected by the evalua 



tion? 

Yes No 

Educational Service Centers X 

Reading Improvement X 

Administrators' Academy X 

Prekindergarten At-Risk X 

Report Card X 

Staff Development X 

Bilingual X 

Gifted/Remedial Summer School X 



A3 V^as the information collected, of sufficient scope and 
selected in such ways to address the RFP questions and 
meet the needs and interests of ISBE? 



Yes Ho 

Educational Service Centers X 
Reading Improvement X 

Administrators' Academy X 

Prekindergarten At-Risk X 
Report Card X 

Staff Development X 
Bilingual X 
Gifted/Remedial Summer School X 



A4 Were the perspectives, procedures, and rationale used 
to interpret the findings carefully described so that 
the basis for value judgments are clear? 



Yes No 

Educational Service Centers X 
Reading Improvement X 
Administrators' Academy X 
Prekindergarten At-Risk X 
Report Card X 
Staff Development X 
Bilingual X 
Gifted/Remedial Summer School X 



ERIC 
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AS Did the evaluation report describe the program and its 
context, the purpose, procedures and findings of the 
evaluation so that the funding agency (ISBE) could readily 
understand what was done, why it was done, what information 
was obtained, what conclusions were drawn, and what 
recommendations were made? 

Yes \io 

Educational Service Centers X 
Reading Improvement X 
Administrators' Academy X 
Prekindergarten At-Risk X 
Report Card X 
Staff Development X 
Bilingual X 
Gifted/Remedial Summer School X 

A6 Were the findings disseminated to ISBE and other right-to- 
know audiences, so that they can assess and use the find- 
ings? 

Yes No 
Educational Service Centers X 
Reading Improvement X 
Administrators' Academy X 
Prekindergarten At-Risk X 
Report Card X 
Staff Development X 
Bilingual X 
Gifted/Remedial Summer School X 

A7 Was the report released in a timely manner so that ISBE 
could best use the reported information? 

Yes No 

Educational Service Centers X 
Reading Improvement X 
Administrators' Academy X 
Prekindergarten At-Risk X 
Report Card X 
Staff Development X 
Cilingual X 
Gifted/Remedial Summer School X 



A8 Was the evaluation planned and conducted in ways that 
encouraged follow-through by ISBE? 



Yes No 

Educational Service Centers X 

Reading Improvement X 

Administrators' Academy X 

Prekindergarten At-Risk X 

Report Card X 

Stafl Development X 

Bilingual X 

Gifted/Remedial Summer School X 



ERIC 
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C3 Was the written evaluation open, direct, and honest in 
its disclosure of pertinent findings, including the 
limitations of the evaluation? 



Yes No 

Educational Service Centers X 

Reading Improvement X 

Administrators' Academy X 

Prekindergarten At-Risk X 
Report Card X 

Staff Development X 
Bilingual X 

Gifted/Remedial Summer School X 



C4 Did the formal parties to the evaluation respect and assure 
the public's right to know, within the limits of other 
related principles and statues, such as those dealing with 
public safety and the right to privacy? 

Yes No 



Educational Service Centers X 

Reading Improvement X 

Administrators' Academy X 

Prekinder^:!arten At-Risk X 

Report Card X 

Staff Development X 

Bilingual X 

Gifted/Remedial Summer School X 



C7 Was the evaluation complete and full in its presentation 
of strengths and weaknesses of the programs so that the 
strengths can be built upon and thp problem areas addressed? 

Yes No 



Educational Service Centers X 

Reading Improvement X 

Administrators' Academy X 

Prekindergarten At-Risk X 
Report Card X 

Staff Development X 

Bilingual X 

Gifted/Remedial Summer School X 
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Dl Was the program sufficiently examined so that the program 

being considered in the evaluation is clearly identified? 

Yes No 
Educational Service Centers X 
Reading Improvement X 
Administrators' Academy X 
Prekindergarten At-Risk X 
Report Card X 
Staff Development X 
Bilingual X 
Gifted/Remedial Summer School X 

D2 Was the context in which the programs are used examined in 
enough detail so that influence on them can be identified? 

Yes No 

Educational Service Centers X 
Reading Improvement X 
Administrators' Academy X 
Prekindergarten At-Risk X 
Report Card X 
Staff Development X 
Bilingual X 
Gifted/Remedial Summer School X 

D3 Was the purposes and procedures of the evaluation monitored 

and described in enough detail so that they could be 
identified and assessed? 

Yes No 
Educational Service Centers X 
Reading Improvement X 
Administrators' Academy X 
Prekindergarten At-Risk X 
Report Card X 
Staff Development X 
Bilingual X 
Gifted/Remedial Summer School X 

D4 Were the sources of information described in enough detail 
so that the adequacy of the information can be assessed? 

Yes No 
Educational Service Centers X 
Reading Improvement X 
Administrators' Academy X 
Prekindergarten At-Risk X 
Report Card X 
Staff Development X 
Bilingual X 
Gifted/Remedial Summer School X 
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DlO Were the conclusions reached in the evaluation explicitly 
justified so that ISBE can assess them? 



Yes No 

Educational Service Centers X 

Reading Improvement X 

Administrators' Academy X 

Prekindergarten At-Risk X 
Report Card X 

Staff Development X 
Bilingual X 

Gifted/Remedial Summer School X 



D 11 Did the evaluation procedures provide si?feguards to 
protect the evaluation findings and report against 
dislocation by one's personal feelings and biases of 



any party to the evaluation. 

Yes No 

Educational Service Centers X 

Reading Improvement X 

Administrators' Academy X 

Prekindergarten At-Risk X 
Report Card X 

Staff Development X 
Bilingual X 

Gifted/Remedial Summer School X 
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